The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
Most current input controls, for example joysticks, voice commands, and gesturing, require the user to locate a point of visual interest on a screen with their eyes. Thus, a user must first identify a view of interest for the robotic device. The user must then formulate appropriate decisions to yield motor responses (through fingers, hands, arms, legs, or speech) to change a particular field of view (FOV) of the robotic device. In most cases, these controls require multi-body-part coordination to manipulate the robotic device, such as a laparoscope, which creates a barrier between the surgeon and the surgical site. Other previous robotic control devices used voice control to effect change in the robotic device. However, studies have demonstrated that voice control is less efficient than a practiced and attentive human assistant.
Fundamentally, the eye is a perceptual organ not meant for an output communication channel. Therefore, using solely gaze as a control input signal requires the differentiation of normal behavioral eye movements and intentional eye “commands,” which is known as the Midas touch problem (the eyes cannot be used directly as a mouse, because the eyes are never “off.”) The “select” or “click” command is usually derived from tedious eye gestures (such as excessive blinks, over-long dwell time, other controlled eye movements) as a confirmation to a specific command from the eye). Ideally, gaze-based robot control would be conducted implicitly without tedious eye gestures.
Existing gaze control modes are indirect, incremental, and offer only relative control of the robotic control (e.g., control with respect to the robot's previous position). This may cause the robotic device to effect direction/joint rotation with only one fixed step, and multiple steps of motion commands are usually needed to approach a target view. Other existing systems adopt regression-based methods, which are highly dependent on the mapping relation between eye movements and a display surface. In the case of visual control of surgical instruments and robotic devices, these methods require a surgeon's head to remain relatively still and a recalibration is needed if the user's head moves. Therefore, regression-based methods are impractical in the operating room or surgical suite as the surgeon may move his/her head and even look at other directions in activities such as tool changing, communication with other personnel, and checking other monitoring instruments (ultrasound image, heartbeat, blood pressure, etc.).
One problem inherent with known laparoscopic surgery techniques is inconvenient, limited visualization in comparison with traditional open surgery. In laparoscopic surgery, the surgeon's view is normally limited to what is in front of the scope. In many cases, the surgeon needs to frequently readjusts the laparoscope
What is needed is a new type of robotic device and control system to make surgeon or other user robotic device interaction more effective and intuitive, and to make the execution of surgical maneuvers smoother and faster. Further, a system which is adaptive the movement of a user's head is desirable to enable freedom of movement. In some cases, described herein, the surgeon's control burden may be reduced by increasing the level of autonomy in the disclosed robotic device, e.g. robotic laparoscope, systems.